Dynamic vision sensor and projector for depth imaging

ABSTRACT

Systems, devices, and techniques related to matching features between a dynamic vision sensor and one or both of a dynamic projector or another dynamic vision sensor are discussed. Such techniques include casting a light pattern with projected features having differing temporal characteristics onto a scene and determining the correspondence(s) based on matching changes in detected luminance and temporal characteristics of the projected features.

BACKGROUND

In computer vision and other imaging and computing contexts, depthimages provide important information for a scene being viewed. A depthimage may be generated based on two (e.g., left and right or referenceand target) two-dimensional images or between a first image and aprojected image. Such applications rely on detecting correspondingfeatures either between two cameras (e.g., two images) or between acamera (e.g., an image) and a projector system (e.g., a projected lightpattern) and performing triangulation. Examples of systems forcorrespondence between camera and projector include coded light camerasand structured light cameras.

A challenge for triangulation based depth cameras is identifying whichfeatures in the camera correspond to the features provided by theprojection system. Overcoming such challenges may require significantcomputational resources, which leads to higher cost and powerconsumption. Another challenge in projection based triangulation systemsis the potential interference from sunlight washing out the projectionpattern. Current systems use complex structured patterns to maximize theunique nature of the projected pattern, especially along the axis of thetriangulation. There is a tradeoff between pattern complexity andpattern size, so the pattern may be repeated over the field of view ofthe camera. Such repeating patterns limit the size of disparity that canbe detected and, therefore, the closest range detectable by the camerasuch that there is a resulting tradeoff between complexity and minimumrange of the camera. Complex processing is used to search for thepatterns along the epipolar axis using a search range that is less thanor equal to the size of the repeating pattern. Sunlight rejection may beaccomplished using a combination of bandpass optical filters that matchthe projector wavelength and synchronizing the pulsing of the laserprojector to the exposure time of a global shutter sensor.

Therefore, current techniques and implementations have limitations. Todetect close objects, the camera must be capable of detecting largedisparities which leads to large, complex patterns and large searchranges thereby increasing the cost of the projector as well as the costand power of required computation resources. Such limitations lead tocomplicated implementation, poor performance in sunlight, and less thandesirable depth map results. It is with respect to these and otherconsiderations that the present improvements have been needed. Suchimprovements may become critical as the desire to utilize depth imagesin a variety of applications becomes more widespread.

BRIEF DESCRIPTION OF THE DRAWINGS

The material described herein is illustrated by way of example and notby way of limitation in the accompanying figures. For simplicity andclarity of illustration, elements illustrated in the figures are notnecessarily drawn to scale. For example, the dimensions of some elementsmay be exaggerated relative to other elements for clarity. Further,where considered appropriate, reference labels have been repeated amongthe figures to indicate corresponding or analogous elements. In thefigures:

FIG. 1 illustrates components of an example system for determiningcorrespondence between pixels of a dynamic vision sensor and featuresprojected from a dynamic projector;

FIG. 2 illustrates components of an example system for determiningcorrespondence between pixels of multiple dynamic vision sensors;

FIG. 3 illustrates an example stereoscopic image matching;

FIG. 4 illustrates exemplary components of the example system of FIG. 1;

FIG. 5 illustrates an example dynamic vision sensor;

FIG. 6 illustrates an example timing diagram for a particular pixel of adynamic vision sensor in operation;

FIG. 7A illustrates a side view of an example dynamic projectorincluding an example VCSEL array;

FIG. 7B illustrates a side view of an example dynamic projectorincluding an example VCSEL array and an example optical element to splitemitted light into predefined patterns;

FIG. 8 illustrates a side view of an example dynamic projector includinga micro-electro-mechanical systems (MEMS) mirror implementation;

FIG. 9 illustrates a depiction of an example light pattern cast on anexample scene;

FIG. 10 illustrates an example signal timing diagram for particularpixels of a dynamic projector, a first dynamic vision sensor, and asecond dynamic vision sensor in operation;

FIG. 11 illustrates an example illumination scheme for use in featurematching;

FIG. 12 illustrates another example illumination scheme for use infeature matching;

FIG. 13 illustrates yet another example illumination scheme for use infeature matching;

FIG. 14 illustrates an example device for determining correspondencebetween a dynamic vision sensor and/or between multiple dynamic visionsensors;

FIG. 15 is a flow diagram illustrating an example process forcorrelating a feature detected at a dynamic vision sensor pixel to afeature of a projected light pattern;

FIG. 16 is an illustrative diagram of an example system for correlatinga feature detected at a dynamic vision sensor pixel to a feature of aprojected light pattern;

FIG. 17 is an illustrative diagram of an example system; and

FIG. 18 illustrates an example small form factor device, all arranged inaccordance with at least some implementations of the present disclosure.

DETAILED DESCRIPTION

One or more embodiments or implementations are now described withreference to the enclosed figures. While specific configurations andarrangements are discussed, it should be understood that this is donefor illustrative purposes only. Persons skilled in the relevant art willrecognize that other configurations and arrangements may be employedwithout departing from the spirit and scope of the description. It willbe apparent to those skilled in the relevant art that techniques and/orarrangements described herein may also be employed in a variety of othersystems and applications other than what is described herein.

While the following description sets forth various implementations thatmay be manifested in architectures such as system-on-a-chip (SoC)architectures for example, implementation of the techniques and/orarrangements described herein are not restricted to particulararchitectures and/or computing systems and may be implemented by anyarchitecture and/or computing system for similar purposes. For instance,various architectures employing, for example, multiple integratedcircuit (IC) chips and/or packages, and/or various computing devicesand/or consumer electronic (CE) devices such as set top boxes, smartphones, etc., may implement the techniques and/or arrangements describedherein. Further, while the following description may set forth numerousspecific details such as logic implementations, types andinterrelationships of system components, logic partitioning/integrationchoices, etc., claimed subject matter may be practiced without suchspecific details. In other instances, some material such as, forexample, control structures and full software instruction sequences, maynot be shown in detail in order not to obscure the material disclosedherein.

The material disclosed herein may be implemented in hardware, firmware,software, or any combination thereof. The material disclosed herein mayalso be implemented as instructions stored on a machine-readable medium,which may be read and executed by one or more processors. Amachine-readable medium may include any medium and/or mechanism forstoring or transmitting information in a form readable by a machine(e.g., a computing device). For example, a machine-readable medium mayinclude read only memory (ROM); random access memory (RAM); magneticdisk storage media; optical storage media; flash memory devices;electrical, optical, acoustical or other forms of propagated signals(e.g., carrier waves, infrared signals, digital signals, etc.), andothers.

References in the specification to “one implementation”, “animplementation”, “an example implementation”, or such embodiments, orexamples, etc., indicate that the implementation, embodiment, or exampledescribed may include a particular feature, structure, orcharacteristic, but every implementation, embodiment, or example may notnecessarily include the particular feature, structure, orcharacteristic. Moreover, such phrases are not necessarily referring tothe same implementation. Furthermore, when a particular feature,structure, or characteristic is described in connection with anembodiment, it is submitted that it is within the knowledge of oneskilled in the art to affect such feature, structure, or characteristicin connection with other implementations whether or not explicitlydescribed herein. The terms “substantially,” “close,” “approximately,”“near,” and “about,” generally refer to being within +/−10% of a targetvalue.

Methods, devices, apparatuses, computing platforms, and articles aredescribed herein related to the use of dynamic vision sensors anddynamic projectors for matching features between the dynamic visionsensors and/or a dynamic projector. Such matched features may be usedfor depth map generation or other applications.

As discussed further herein, a system according to some embodimentsincludes a dynamic vision sensor based camera and a dynamic patternprojector. As used herein, a dynamic vision sensor (or dynamic imagesensor) is any image sensor that detects and indicates events at pixelsof the sensor as pixel(s) change (e.g., due to illumination intensitychange at the pixel). Such change(s) at pixel(s) of the dynamic visionsensor may be change(s) in luminance over time, for example. Forexample, the change(s) may be detected at individual pixels or at setsof pixels. As used herein, a set of pixels includes two or more adjacentpixels. Dynamic vision sensors may be contrasted with traditional imagesensors as follows. Traditional image sensors detect luminance at aphotodiode over a predetermined duration (i.e., an exposure time) ascharge is collected in the pixel circuit. Such a traditional imagesensor then transmits information from each pixel sensor for theexposure as a frame or picture. The traditional image sensor maytransmit complete frames at a predetermined or fixed frame rate forexample. For example, traditional image sensors apply a reset voltage toa photodiode to drain any charge in parasitic capacitance of a depletionregion and then sample the charge at the end of a fixed integrationperiod. The amount of charge that can be held in the photodiodecapacitance limits the dynamic range of the sensor. In contrast, as usedherein, a dynamic vision sensor, does not rely on charge collection butinstead detects and indicates temporal changes at pixels of the sensor.In an embodiment, a dynamic vision sensor directly measures thephoto-current in the photodiode. Changes in the photo-current over timeare indicated as events (e.g., ON and OFF or increase/decrease events,etc.) at the corresponding pixel. Such sensors offer high dynamic rangesince photodiode capacitance does not place a constraint duringoperation. In an embodiment, in a dynamic vision sensor pixel, thephotocurrent is converted to a voltage that is fed to an amplifieddifferencer circuit, which enables detection in increases or decreasesin the photocurrent. In a dynamic vision sensor, whenever a pixelincreases in intensity, the sensor generates an output and resets thereference based on the new intensity value. The dynamic range of adynamic vision sensor is limited by the maximum voltage that thephotodiode can generate (in comparison with its noise) and dynamicvision sensors may achieve 120 dB (e.g., 6 decades) of dynamic range,which is suitable for outdoor, nighttime applications for example.

Such dynamic vision sensors provide significant improvements in temporalresolution (e.g., about 1-10 μs in contrast to 10-17 ms for traditionalimage sensors). As discussed further herein, an imaging device includesa dynamic vision sensor and a dynamic projector (or active projector) toprovide pixel correlation based on a scene. The pixel correlation may bebetween a dynamic vision sensor and a dynamic projector or betweenmultiple dynamic vision sensors (based on a light pattern provided bythe dynamic projector). By controlling the temporal behavior of thelight pattern provided by the dynamic projector, temporal correspondenceis leveraged to identify corresponding features of a scene (e.g., afeature detected in a sensor that corresponds to either a projectedfeature or the same feature in a second sensor).

As discussed, such temporal correspondence may be detected between adynamic vision sensor and a dynamic projector or between multipledynamic vision sensors (based on a light pattern provided by the dynamicprojector). In embodiments where a dynamic vision sensor and a dynamicprojector are implemented, a known temporal characteristic or signatureis provided for a projected feature of the light pattern by a pixel ofthe dynamic projector. A match of the known temporal characteristic orsignature is detected at a particular pixel (or multiple adjacentpixels) of the dynamic vision sensor. Thereby, a correspondence isdetermined between the pertinent feature of the dynamic projector andthe pertinent feature of the dynamic vision sensor. Such correspondencemay be used for a variety of imaging or computer vision techniques. Forexample, triangulation may be used to determine a depth value ordisparity value for a point or region of a scene.

In embodiments where multiple dynamic vision sensors and a dynamicprojector are implemented, a known temporal characteristic or signatureis again provided for a projected feature of the light pattern by apixel of the dynamic projector. The known temporal characteristic orsignature is detected at a particular pixel (or multiple adjacentpixels) of a first dynamic vision sensor and a particular pixel (ormultiple adjacent pixels) of a second dynamic vision sensor. Thereby, acorrespondence is determined between the pertinent features of the firstand second dynamic vision sensors. Again, such correspondence may beused for a variety of imaging or computer vision techniques such astriangulation to determine a depth value or disparity value for a pointor region of a scene. Although discussed with respect to embodimentswith a dynamic vision sensor and a dynamic projector and two dynamicvision sensors and a dynamic projector, any number of dynamic projectorsand dynamic vision sensors may be used for improved correspondence,calibration, etc.

FIG. 1 illustrates components of an example system 100 for determiningcorrespondence between pixels of a dynamic vision sensor and featuresprojected from a dynamic projector, arranged in accordance with at leastsome implementations of the present disclosure. As shown in FIG. 1,system 100 may include an image signal processor (ISP) 101 to implementa feature matching module 102 and a light pattern module 103 (labeledlight pattern with temporal variation), a dynamic vision camera (DVC)104, a dynamic projector (DP) 105, a driver 108, a memory 107, and acomputer vision module 106. In an embodiment, dynamic projector 105 anddynamic vision camera 104 are aligned along a first axis (e.g.,horizontally along the x-axis as illustrated) such that matched featuresmay be used to generate depth or disparity values for scene 121. Forexample, the axis between dynamic projector 105 and dynamic visioncamera 104 may define an epipolar axis. However, dynamic projector 105and dynamic vision camera 104 may be provided at any suitableorientation.

Also as shown, dynamic projector 105 projects a light pattern 114 onto ascene 121. Dynamic projector 105 may project light pattern 114 using anysuitable type of light, wavelength range, etc. such that light pattern114 is detectable by dynamic vision camera 104. In an embodiment,dynamic projector 105 projects an infrared (IR) light pattern 114. SuchIR light patterns may be advantageous since they will not distract usersof system 100 or people in scene 121. Light pattern 114 may include anysuitable spatial pattern such as a grid pattern or the like (pleaserefer to FIG. 9). Light pattern 114 also includes temporal patternsamong spatial components (e.g., projected features) thereof such thatcorrespondence between features of dynamic projector 105 and dynamicvision camera 104 may be determined using the temporal patterns.

Dynamic vision camera 104 may include any suitable dynamic vision sensorto sense pixel based temporal changes as discussed herein. Furthermore,dynamic projector 105 may include any suitable dynamic projector toprovide light pattern 114 having projected features with differingtemporal characteristics. As shown, dynamic projector 105 projects lightpattern 114 onto scene 121. Dynamic vision camera 104 detects incomingillumination 115 as reflected off of scene 121 such that illumination115 includes reflectance of light pattern 114. Projected features oflight pattern 114 including projected feature 122, which may be a dot orother shape of illumination onto scene 121, have differing temporalcharacteristics or signals such that reflection 123 from projectedfeature 122 has the same or substantially the same temporalcharacteristics. Such temporal characteristic(s) may include, forexample, a turn on time, a turn off time, an illumination duration,multiple illumination events (e.g., each illumination event including anon time, illumination duration, off time) of the same or varyingfrequencies, etc. Furthermore, such temporal characteristic(s) may beunique to only projected feature 122, unique to projected features(including projected feature 122) orthogonal to an epipolar axis betweendynamic projector 105 and dynamic vision camera 104, or unique toprojected features within a region of scene 121 (e.g., a half of scene121), or the like. In any event, the temporal characteristics ofprojected feature 122 (and reflection 123) are used to establish acorrespondence between projected feature 122 (and/or the pixel oremitter of dynamic projector 105 casting projected feature 122 ontoscene 121) and the pixel (or group of pixels) of dynamic vision camera104 detecting reflection 123. Projected features as discussed herein maybe characterized as projected components, illumination components,projected elements, illumination elements, or the like.

For example, light pattern module 103 provides a signal indicating lightpattern 114 (including differing temporal characteristics amongcomponents thereof) to driver 108, which may translate the signal tocontrol signal 111, which provides low level signaling to dynamicprojector 105 for the output of light pattern 114 over time. Dynamicvision camera 104, via a dynamic vision sensor as discussed herein,senses illumination 115 over time and provides pixel signals 112. Pixelsignals 112 may include any suitable data structure or the like suchthat pixel signals 112 indicate pixel changes for pixels of the dynamicvision sensor over time. For example, pixel signals 112 may include, foreach pixel change (increase in illumination greater than a particularamount over time, decrease in illumination greater than a particularamount over time, etc.) a pixel location and temporal characteristicsfor the change. The temporal characteristics may include a time stampfor the change, whether the change was an increase or decrease, aduration of the change, or other temporal information.

Also as shown, light pattern module 103 provides an illumination patternsignal 118 to feature matching module 102. Illumination pattern signal118 and the signal (not labeled) provided to feature matching module 102from light pattern module 103 may be the same or they may be different.Illumination pattern signal 118 indicates characteristics of lightpattern 114 pertinent to feature matching module 102 performing featurematching based on illumination pattern signal 118 and pixel signals 112from dynamic vision camera 104. For example, illumination pattern signal118 may include any suitable data structure or the like such thatillumination pattern signal 118 indicates features of dynamic projector105 and the temporal characteristics thereof such that matches may bemade between illumination pattern signal 118 and pixel signals 112. Forexample, illumination pattern signal 118 may include, for each pixelchange to be implemented by dynamic projector 105 (turn on, illuminationduration, turn off, etc.) a pixel location and a time stamp for thepixel change such that a match between the temporal characteristicimplemented by a pixel of dynamic projector 105 may be matched to atemporal characteristic indicated by pixel signals 112.

For example, one or more of a turn on time stamp, an illuminationduration, a turn off time stamp, etc. implemented in projected feature122 via dynamic projector 105 may be matched to one or more of a turn ontime stamp, an illumination duration, a turn off time stamp, etc.detected by a pixel of dynamic vision camera 104 as indicated by pixelsignals 112 to indicate a match of temporal characteristics. As will beappreciated, temporal information such as one or more of a turn on timestamp, an illumination duration, a turn off time stamp, etc. may matchwithin the granularity of data provided by illumination pattern signal118 and pixel signals 112 due to the rate of travel of projected feature122; however, some variation may be resolved using thresholding, signaldelay processing, or other techniques.

Furthermore, illumination pattern signal 118 includes the location(i.e., the feature's coordinates in angular space) of dynamic projector105 used to generate projected feature 122 (i.e., the projected angle ofprojected feature 122) having the pertinent temporal characteristics andpixel signals 112 indicates the location (i.e., the pixel location) ofthe dynamic vision sensor of dynamic vision camera 104 having thematching temporal characteristics. Thereby, feature matching module 102has a correspondence between an angular coordinate of dynamic projector105 and a pixel of dynamic vision camera 104 for a position (x) of scene121. Such correspondence may be used in a variety of image processing orcomputer vision contexts. For example, feature matching module 102 maydetermine a depth value of depth map 113 (or a disparity value of adisparity map) using the correspondence and the spatial orientations ofdynamic vision camera 104 and dynamic projector 105. For example,feature matching module 102 may determine a depth value of depth map 113(or a disparity value of a disparity map) using the projected angle ofprojected feature 122 (e.g., an angle in the x-y plane between projectedfeature 122 and the z-direction normal to dynamic projector 105), thelocation of the pixel(s) of the dynamic image sensor (i.e., as based onthe feature correspondence) and the spatial orientations of dynamicvision camera 104 and dynamic projector 105.

In an embodiment, such temporal signaling and detection techniques arerepeated for a range, array, or grid of positions of scene 121 byproviding any number of projected features in analogy to projectedfeature 122, detecting a match for the temporal characteristic at apixel or pixels of the dynamic vision sensor of dynamic vision camera104 to establish a correspondence between a dynamic projector 105feature and a feature of dynamic vision camera 104, and, optionally,performing triangulation to determine a depth value of depth map 113 (ora disparity value of a disparity map) using the correspondence. Eachprojected feature 122 may be provided in series, partially or fully inparallel, etc. such that unique correspondences may be detected. Suchtemporal characteristics of light pattern 114 are discussed furtherherein below.

As shown, in an embodiment, depth map 113 (or a disparity map) may bestored to memory 107 and accessed by computer vision module 106. In anembodiment, feature matching module 102 generates feature correspondencedata or information (e.g., without generating a depth or disparity map)that include a data structure indicating corresponding features (i.e.,features that match or correspond to one another due to having matchingtemporal characteristics), corresponding depth vision sensor pixels andprojected angles, corresponding pixels of dynamic projector 105 and thedynamic vision sensor of dynamic vision camera 104, or the like. Suchcorrespondence information may be stored to memory 107 for subsequentimage processing, computer vision procession, or the like. In anembodiment, ISP 101 includes a separate depth map module that uses suchcorrespondence information to generate depth map 113 (or a disparitymap).

System 100, system 200 discussed below, or any combination of componentsthereof may be implemented via any suitable device such as a depthsensor, a depth sensor module, an imaging device, a personal computer, alaptop computer, a tablet, a phablet, a smart phone, a digital camera, agaming console, a wearable device, a set top device, a drone or thelike.

FIG. 2 illustrates components of an example system 200 for determiningcorrespondence between pixels of multiple dynamic vision sensors,arranged in accordance with at least some implementations of the presentdisclosure. As shown in FIG. 2, system 200 may include any of thecomponents discussed with respect to system 100 along with a seconddynamic vision camera (DVC) 201. For example, in the embodiment ofsystem 100, pixel correspondence is determined between dynamic projector105 and dynamic vision camera 104. In the embodiment of system 200,pixel correspondence is determined between dynamic vision camera 104 anddynamic vision camera 201. In addition, correspondence may also bedetermined between dynamic projector 105 and one or both of dynamicvision cameras 104, 201 of system 200. In an embodiment, dynamic visioncameras 104, 201 are aligned along a first axis (e.g., horizontallyalong the x-axis as illustrated) such that matched pixels may be used togenerate depth or disparity values for scene 121. For example, the axisbetween dynamic vision cameras 104, 201 may define an epipolar axis. Inan embodiment, dynamic projector 105 is also aligned horizontally withrespect to dynamic vision cameras 104, 201.

However, dynamic projector 105 may be provided at any suitableorientation. In some embodiments, dynamic projector 105 is aligned offaxis with respect to dynamic vision cameras 104, 201. Such embodimentsmay provide for calibration recovery after a drop of system 200 oranother event that provides for a loss of calibration for system 200,for example. In an embodiment, dynamic vision cameras 104, 201 anddynamic projector 105 are aligned in the shape of a triangle such thatseparate epipolar axes are formed between each dynamic vision camera anddynamic projector pair. For example, dynamic projector 105, dynamicvision sensor 104, and dynamic vision sensor 201 may be configured in atriangular shape such that dynamic projector 105 is off-axis andorthogonal to an axis between dynamic vision sensor 104 and dynamicvision sensor 201. For example, dynamic vision sensor 104 and dynamicvision sensor 201 may have an axis therebetween (e.g., in thex-direction) and dynamic projector 105 may be between them and off-axis(e.g., between them in the x-direction and offset in the y-direction).In an embodiment, dynamic vision cameras 104, 201 are part of a grid ofdynamic vision cameras such that the grid includes 9, 16 or more dynamicvision cameras.

As discussed with respect to dynamic vision camera 104, dynamic visioncamera 201 may include any suitable dynamic vision sensor to sense pixelbased temporal changes. As discussed, dynamic projector 105 projectslight pattern 114 onto scene 121. Dynamic vision cameras 104, 201 detectincoming illumination 115, 215, respectively, as reflected off of scene121 such that illumination 115, 215 includes reflectance of lightpattern 114. As discussed, projected features of light pattern 114including projected feature 122 have differing temporal characteristicsor signals such that reflections 123, 223 from projected feature 122 hasthe same or substantially the same temporal characteristics. Thematching temporal characteristics of reflections 123, 223 are then usedto establish a correspondence between a feature (or group of pixels) ofdynamic vision camera 104 detecting reflection 123 and a feature (orgroup of pixels) of dynamic vision camera 201 detecting reflection 223.

For example, light pattern module 103 provides a signal indicating lightpattern 114 (including differing temporal characteristics amongcomponents thereof) to driver 108, which may translate the signal tocontrol signal 111, which provides low level signaling to dynamicprojector 105 for the output of light pattern 114 over time. Dynamicvision cameras 104, 201, via dynamic vision sensors therein, sensesillumination 115, 215, respectively, over time and provide pixel signals112, 212, respectively, to feature matching module 102 or another moduleof image signal processor 101. As discussed with respect to pixelsignals 112, pixel signals 212 may include any suitable data structureor the like such that pixel signals 212 indicate pixel changes forpixels of the dynamic vision sensor of dynamic vision camera 201 overtime. For example, pixel signals 212 may include, for each pixel change(increase in illumination greater than a particular amount over time,decrease in illumination greater than a particular amount over time,etc.) a pixel location and temporal characteristics for the change. Thetemporal characteristics may include a time stamp for the change,whether the change was an increase or decrease, a duration of thechange, or other temporal information.

As shown, pixel signals 112 and pixel signals 212 are provided tofeature matching module 102. Feature matching module 102 determinesmatches between temporal characteristics as indicated between pixelsignals 112 and pixel signals 212. For example, one or more temporalcharacteristics are matched between pixel signals 112 and pixel signals212. Such matching characteristics may be any suitable temporalcharacteristics such as a time stamp of increased illumination (e.g.,two pixel locations having increased luminance at the same time), a timestamp of decreased illumination (e.g., two pixel locations havingdecreased luminance at the same time), multiple matching ON/OFF eventsof matching frequency (e.g., two pixel locations having matchingincreased and decreased luminance events at the same time), or anycombination thereof. As will be appreciated, temporal information suchas time stamps of increased or decreased illumination, duration, etc.may match within the granularity of data provided by pixel signals 112,212 due to the rate of travel of projected feature 122; however, somevariation may be resolved using thresholding, signal delay processing,or other techniques.

Furthermore, pixel signals 112 indicate the feature (i.e., the pixellocation) of the dynamic vision sensor of dynamic vision camera 104 andthe feature (i.e., the pixel location) of the dynamic vision sensor ofdynamic vision camera 201 having the matching temporal characteristics.Thereby, feature matching module 102 has a correspondence between apixel location of dynamic vision camera 104 and a pixel location ofdynamic vision camera 201 for a position (x) of scene 121. Suchcorrespondence may be used in a variety of image processing or computervision contexts. For example, feature matching module 102 may determinea depth value of depth map 113 (or a disparity value of a disparity map)using the correspondence. Such temporal signaling and detectiontechniques may be repeated for a range, array, or grid of positions ofscene 121 by providing any number of projected features in analogy toprojected feature 122, detecting a match for the temporal characteristicof a feature detected by the dynamic vision sensor of dynamic visioncamera 104 and a feature detected by the dynamic vision sensor ofdynamic vision camera 201, and, optionally, performing triangulation todetermine a depth value of depth map 113 (or a disparity value of adisparity map) using the correspondence. Each projected feature 122 maybe provided in series, partially or fully in parallel, etc. such thatunique correspondences may be detected.

As discussed with respect to system 100, depth map 113 (or a disparitymap) may be stored to memory 107 and accessed by computer vision module106 or feature matching module 102 may generate correspondenceinformation such as a data structure indicating corresponding features(i.e., features that match or correspond to one another due to havingmatching temporal characteristics) between the dynamic vision sensor ofdynamic vision camera 104 and the dynamic vision sensor of dynamicvision camera 201. Such correspondence information may be stored tomemory 107 for subsequent image processing, computer vision procession,or the like.

FIG. 3 illustrates an example stereoscopic image matching 300, arrangedin accordance with at least some implementations of the presentdisclosure. As shown in FIG. 3, stereoscopic image matching 300 mayinclude attaining first and second (or left and right) matching features(as indicated by features x_(L) and x_(R) within image planes 314 a and314 b for the sake of clarity of presentation). As discussed, matchingfeatures may be determined by matching pixels (e.g., locations ofpixels) between dynamic vision cameras 104, 201 using temporalcharacteristics. As shown, scene 121 may include an example surface 310.Scene 121 may include any suitable scene including indoor or outdoorscenes, scenes including objects and/or people, and so on. Stereomatching techniques may determine a depth image based on triangulatingcorresponding features (and/or the locations of corresponding pixelsbetween the dynamic vision sensors). For example, as shown in FIG. 3,given left and right image planes 314 a and 314 b, each including arepresentation of three-dimensional point x on surface 310, the depth,d, of x, may be determined based on d=f*b/disp, where f and b are thefocal length and base line, respectively, and disp, is the disparity forx, indicating the pixel displacement of x between the pertinent pixel ofthe dynamic vision sensor of dynamic vision camera 104 and thecorresponding (matching) feature of the dynamic vision sensor of dynamicvision camera 201. Furthermore, the baseline, b, dynamic vision camera104 to dynamic vision camera 201 provides an epipolar line or axis. Asis discussed further herein, temporal characteristic(s) of projectedfeatures are varied across scene 121 to provide correspondence betweenfeatures detected by dynamic vision cameras 104, 201. In someembodiments, temporal characteristic(s) of projected features may be thesame along the y-axis (e.g., orthogonal to the discussed epipolar axisor plane) since pixel matching is not performed along the y-axis but isinstead restricted to the matching along the x-axis. For example, suchprojected features may be projected by pixels that are orthogonal to anepipolar axis between the dynamic projector and the dynamic visionsensor and/or such projected features may be projected at angles thatare orthogonal to the epipolar axis.

Although illustrated with respect to system 200, referring again to FIG.1, depth values may be generated using similar techniques by replacingthe pixel displacement of x between the pertinent feature of the dynamicvision sensors of dynamic vision cameras 104, 201 with the pixeldisplacement of x between the pertinent feature of the dynamic visionsensor of dynamic vision camera 104 and the corresponding (matching)feature of dynamic projector 105 and/or with a location of the pixels inone or both of dynamic vision cameras 104, 201 and a projected anglefrom the dynamic projector. For example, the projected angle may be anangle between the positive z-direction and the projected feature takenin the x-z plane. However, any suitable projected angle may be used.

FIG. 4 illustrates exemplary components of example system 100, arrangedin accordance with at least some implementations of the presentdisclosure. In particular, FIG. 4 illustrates an example dynamic visioncamera 104 and an example dynamic projector 105. As shown, dynamicvision camera 104 may include a lens 401 adjacent to a dynamic visionsensor 402, both of which are implemented within a housing 403. Lens 401may be moveable to focus illumination 115 onto dynamic vision sensor402. Dynamic vision camera 104 may further include circuitry or the liketo generate pixel signals 112 based on received illumination such asillumination 115 (please refer to FIG. 1).

FIG. 5 illustrates an example dynamic vision sensor 402, arranged inaccordance with at least some implementations of the present disclosure.As shown in FIG. 5, dynamic vision sensor 402 includes a grid of sensorspixels 501. Sensors pixels 501 may include any suitable circuitry andmaterials that may provide for the detection of a change in illuminationover time and the indication of such a change. As discussed, a set ofsensor pixels 501 are pixels that are adjacent to at least one otherpixel in the set. As shown, in an embodiment, an exemplary sensor pixel501 includes a photoreceptor 511, a differencing circuit 512 (includinga reset switch 513), and comparators 514. Incident light 502 is detectedby photoreceptor 511 (e.g., including a photodiode) and photo-currentfrom photoreceptor 511 is converted to a voltage that is fed todifferencing circuit 512 (e.g., an amplified differencer circuit) thatenables detection in increases or decreases in the photocurrent viacomparators 514. An increase in photocurrent is detected as an ON event515 and a decrease in photocurrent is detected as an OFF event 516. Forexample, sensor pixels 501 do not rely on charge collection (as intraditional image sensors) and instead directly measure photo-current inphotoreceptor 511. Such a configuration offers advantageously highdynamic range in contrast to traditional sensor pixels since photodiodecapacitance does not provide a constraint. Sensor pixels 501 of dynamicvision sensor 402 may attain 120 dB of dynamic range in some examples.

FIG. 6 illustrates an example timing diagram 600 for a particular pixelof a dynamic vision sensor in operation, arranged in accordance with atleast some implementations of the present disclosure. As shown in FIG.6, as the light level (e.g., illumination or intensity) at a particularpixel, as indicated by the absolute intensity of example pixel intensity601, increases or decreases over time, ON events 602 (e.g.,corresponding to absolute intensity increases) and OFF events 603 (e.g.,corresponding to absolute intensity decreases) are detected. Herein, OFFevents are indicated using a hatched fill and ON events are indicatedwithout use of a fill. As shown with respect to ON events 604, whenthere is a sharp increase in light level, many or several ON events aregenerated in rapid succession. Thereafter, the circuit settles andreturns to generating ON and OFF events as triggered by changes in theabsolute intensity. In some embodiments, ON and OFF events 602, 603included in pixel signals 112, 212. For example, pixel signals 112, 212may include information of the pixel corresponding to timing diagram 600(e.g., an x, y pixel location or other identifier) and informationindicating ON and OFF events 602, 603 such as a time stamp of the eventand whether the event is an ON or OFF event. In embodiments wherematching is performed between dynamic vision sensor, such time stamps orother data representing ON and/or OFF events may be matched betweenpixel signals 112, 212 as discussed and the pixels corresponding theretomay be defined as matching or corresponding pixels for the purposes ofdepth image processing or the like. In embodiments where matching isperformed between a dynamic vision sensor and a dynamic projector, suchtime stamps or other data representing ON and/or OFF events from pixelsignals 112 may be matched to the same or similar information or dataprovided by illumination pattern signal 118. For example, illuminationpattern signal 118 may have a data format that matches pixel signals112, which in turn may be defined by ON and OFF events 602, 603 asgenerated by sensor pixels 501.

Returning to FIG. 4, as discussed, dynamic vision camera 104 includinglens 401 and dynamic vision sensor 402 within housing 403 may beimplemented in system 100. The same or similar components may also beimplemented as dynamic vision camera 201 as discussed with respect tosystem 200. Also as shown in FIG. 4, in an embodiment, dynamic projector105 of system 100 or system 200 may include an array of vertical-cavitysurface-emitting lasers (VCSELs), VCSEL array 406, adjacent to a lens405. VCSEL array 406 and lens 405 may also optionally be implementedwithin a separate housing (not shown) or within housing 403. VCSEL array406, under the control of driver 108 (illustrated as a projector driverin FIG. 4), emits light emission 407, which may be focused by lens 405to generate light pattern 114 as discussed herein.

FIG. 7A illustrates a side view of an example dynamic projectorincluding an example VCSEL array 406, arranged in accordance with atleast some implementations of the present disclosure. As shown in FIG.7A, VCSEL array 406 includes an array or grid of VCSELs including VCSEL702 disposed on/and or within a substrate 701. For example, VCSEL array406 may be arranged on substrate 701 in a grid pattern in the x-y plane.The embodiment of FIG. 7A may provide a dynamic projector that providesa 1:1 correspondence between each VCSEL of VCSEL array 406 and projectedfeatures (e.g., beams, points, or dots) projected onto scene 121. Asshown, each VCSEL 702 is assembled into a 2D array over substrate 701and each VCSEL 702 emits, under control of driver 108, through lens 405(e.g., a projector lens) to cast a corresponding projected feature ontoscene 121. For example, each VCSEL 702 may represent a row of VCSELsextending in the y-axis along substrate 701.

In an embodiment, each VCSEL 702 is independently controllable toseparately provide a projected feature having a particular temporalcharacteristic or characteristics as discussed herein. In anotherembodiment, each row of VCSELs 702 grouped together along the y-axis isindependently (i.e., on a row-by-row basis) controllable to provideprojected features having a particular temporal characteristic orcharacteristics such that the characteristics match along the y-axis. Insuch embodiments, the y-axis is arranged orthogonal to the epipolar axisof triangulation (please refer to FIGS. 1, 2, and 3) such that thetemporal characteristic or characteristics not being unique along they-axis does not hinder matching pixels along the epipolar axis (e.g.,along the x-axis). For example, the temporal characteristic orcharacteristics of projected features may be unique along the x-axis butnot unique along the y-axis.

FIG. 7B illustrates a side view of an example dynamic projectorincluding example VCSEL array 406 and an example optical element 703 tosplit emitted light into patterns 704, arranged in accordance with atleast some implementations of the present disclosure. As shown in FIG.7B, optical element 703 may be disposed on or over VCSEL array 406 suchthat the emitted light from each of VCSELs 702 is split into sets orgroups of predefined patterns 704 such that each of patterns 704includes multiple projected features or illumination components in adefined pattern. For example, using optical element 703, an entirepattern (e.g., projected feature) may be generated by each of VCSELs702. Optical element 703 may be any suitable optical element or includeany suitable optical elements that split emitted light from each ofVCSELs 702 into a set or group of projected features to define patterns704. In an embodiment, optical element 703 is a diffractive opticalelement. In an embodiment, optical element 703 is a microlens ormultiple microlenses. In an embodiment, optical element 703 is ametasurface or metamaterial. Patterns 704 may be the same for each ofVCSELs 702 or they may be different. Furthermore, each of patterns 704of projected features may be any suitable patterns of any predefinedprojected features such as dots or other shapes. In an embodiment,patterns 704 provide projected features that are orthogonal to anepipolar axis as discussed herein.

FIG. 8 illustrates a side view of an example dynamic projector includinga micro-electro-mechanical systems (MEMS) mirror implementation,arranged in accordance with at least some implementations of the presentdisclosure. For example, dynamic projector 105 or any other dynamicprojector discussed herein may implement the components of FIG. 8 toprovide light pattern 114 having projected features with differingtemporal characteristics as discussed herein. As shown in FIG. 8,dynamic projector 105 may include an edge laser 803 and a 2D MEMSscanner 802 mounted to a substrate 801. Additional optical elements mayalso be employed in the optical path (e.g., diffusers, integrators, andcollimators). In an embodiment, MEMS scanner 802 is configured to scanover both a first projection angle (e.g., vertically) and an orthogonalsecond projection angle (e.g., horizontally) to vary the illuminationpoint location along a first axis (e.g., y-axis) and second axis (e.g.,x-axis) in response to an electrical signal applied thereto. Forexample, MEMS scanner 802 may scan in response to control signal 111provided by driver 108. Light from edge laser 803 may be temporallycontrolled (e.g., pulsed) as MEMS scanner scans to generate lightpattern 114 having varying temporal characteristics as discussed herein.For example, edge laser 803 and MEMS scanner 802 may operate in concertto provide light pattern 114 having varying temporal characteristicsacross scene 121.

In another embodiment, MEMS scanner 802 is configured to scan only inone projection angle (e.g., horizontally along the x-axis) while beingfixed in the other angle (e.g., vertically or along the y-axis). Forexample, MEMS scanner 802 may scan in the x-direction in response tocontrol signal 111 provided by driver 108. Furthermore, light from edgelaser 803 may be temporally controlled (e.g., pulsed) as MEMS scannerscans to generate light pattern 114 having varying temporalcharacteristics as discussed herein. As discussed, in some embodiments,pixel matching is performed only in the x-direction and having temporalcharacteristics that do not vary along the y-axis (e.g., the epipolaraxis of triangulation) does not hinder pixel matching in thex-direction. Indeed, such characteristics may reduce scan times, pixelmatching times, etc.

Although discussed with respect to a VCSEL array projector embodiment(e.g., FIG. 7) and a MEMS scanner projector embodiment (e.g., FIG. 8),any suitable dynamic projector may be implemented to generate lightpattern 114. In some embodiments, an emitter of a dynamic projectorincludes a mirror of a deformable micro-mirror device (DMD). Forexample, a DMD may include a projection array having a 100×100 mirrorarray, a 1000×1000 mirror array, or more. In addition to the projectionarray, such a projector may further include one or more edge-emitting(or surface emitting) laser light sources configured to shine on all, ora portion of, the mirror array.

FIG. 9 illustrates a depiction of an example light pattern 901 cast onan example scene 900, arranged in accordance with at least someimplementations of the present disclosure. For example, FIG. 9illustrates an example light pattern 901 with each pixel location of adynamic projector such as dynamic projector 105 illuminatedsimultaneously to generate each available projected feature 904 for thepurposes of clarity of presentation. For example, light pattern 114 mayilluminate projected features 904 with various temporal characteristicsas discussed herein. For example, scene 900 may be illuminated, at anytime instance, by one or all or any combination of projected features904 of light pattern 901. In practice, some or all of projected features904 having varying temporal characteristics such that pixel matches maybe detected using such varying temporal characteristics of projectedfeatures 904. For example, one or more dynamic vision sensors detecttemporal changes of projected features 904 as pixel responses thereinand such responses may be correlated between dynamic vision sensorsand/or between a dynamic vision sensor and the dynamic projector used togenerate the temporal changes of projected features 904 to provide pixelmatches.

As shown in FIG. 9, scene 900 includes a light pattern 901 (i.e., whitedots in the illustration) from light pattern 114 being projected or castonto scene 121. As shown, in an embodiment, light pattern 901 may have agrid like pattern of projected feature 904 dots, beams, specks, or thelike. However, light pattern 901 may have any suitable pattern such as arandom speckle pattern, a concentric ring pattern, etc. of projectedfeature 904 shapes projected shapes such as squares, rectangles,diamonds, or the like. Scene 900 may include any suitable scene. In theillustrated embodiment, scene 900 includes a foreground object 903(e.g., a table) and a background 902. In some embodiments, it isadvantageous to determine a depth map of scene 900 such that the depthmap indicates depths of objects as they are positioned with scene 900.

FIG. 10 illustrates an example signal timing diagram 1000 for particularpixels of a dynamic projector, a first dynamic vision sensor, and asecond dynamic vision sensor in operation, arranged in accordance withat least some implementations of the present disclosure. As shown inFIG. 10, a pixel of a dynamic projector may provide, for a particularprojected feature, an illumination intensity signal 1001 such that thepixel (and corresponding projected feature) is initially off, is turnedon at time t0, and is turned off at time t1 (such that the illuminationevent has an illumination duration of t1−t0). As shown, in response toillumination intensity signal 1001 a pixel (or multiple adjacent pixels)of a first dynamic vision sensor reacts with an event signal 1002indicating an ON event at time t0 and an OFF event at time t1 such thatthere is a corresponding illumination duration determined as the timedifference between the ON event and the OFF event (e.g., t1−t0).Similarly, a second dynamic vision sensor reacts with an event signal1003 indicating an ON event at time t0 and an OFF event at time t1 and acorresponding illumination duration (e.g., t1−t0).

As will be appreciated, prior to signal analysis, the only known pixelcorresponding to signals 1001, 1002, 1003, is the pixel of the dynamicprojector emitting illumination intensity signal 1001 via thecorresponding projected feature. Upon receiving one or both of eventsignals 1002, 1003, feature matching module 102 or similar componentdetermines which pixel of the first dynamic vision sensor has an eventsignal that matches illumination intensity signal 1001 (for matchingbetween dynamic projector and dynamic vision sensor) and/or event signalmatches between pixels of the first dynamic vision sensor and the seconddynamic vision sensor.

Such matching may be performed using any suitable technique ortechniques. For embodiments that match pixels of a dynamic projector andpixels of a dynamic vision sensor, pixel signals such as event signal1002 (e.g., as provided by pixel signals 112) for some or all pixels ofthe dynamic vision sensor are monitored for one or more of an ON eventat time t0 (or within a threshold of time t0), an OFF event at time t1(or within a threshold of time t1), and difference between temporallyadjacent ON and OFF events of t1−t0. For embodiments that match pixelsbetween dynamic vision sensors, pixel signals such as event signal 1002(e.g., as provided by pixel signals 112) and event signal 1003 (e.g., asprovided by pixel signals 212) for some or all pixels of the dynamicvision sensors are monitored for one or more ON events that are at thesame time t0 (or within thresholds of time t0), OFF events that are atthe same time t1 (or within a thresholds of time t1), and differencesbetween temporally adjacent ON and OFF events of t1−t0 (or within athreshold of the difference t1−t0). Thereby, matching pixels may bedetermined and such correspondences may be saved to memory, used fordepth map generation, and/or other computer vision or imagingapplications. As will be appreciated, signals 1001, 1002, 1003illustrates substantially perfect timing therebetween. However, inpractice, lag or signaling loss may be mitigated using thresholdingtechniques, signal analysis techniques, etc.

FIG. 10 illustrates relatively simple signals 1001, 1002, 1003 for thesake of clarity of presentation. Such signals may have any suitablecharacteristics as driven by illumination intensity signal 1001. Forexample, illumination intensity signal 1001 may include multiple on andoff events of the same or differing intensities with the same ordiffering durations having the same or different times therebetween(e.g., in the off state), and the like. Furthermore, such temporalcharacteristics (e.g., turning a projected feature on at a particularstart time, holding on for a particular duration, turning off at aparticular end time, and optionally repeating with the same duration,differing duration, intensity, frequency, etc.) may be provided to onlyone projected feature or to multiple projected features simultaneouslysuch that pixel correspondence may be provided as discussed herein.

With reference to FIG. 9, a simplistic approach may be to turn on andoff each of projected features 904 in turn in the same manner (e.g.,using the same temporal characteristics except for start and stop times)and to monitor the pixels of either one or both of dynamic visioncameras to determine correspondence. While such an approach may offerease of implementation, it may be inefficient in terms of detectiontime, depth map generation time, etc. Instead, differing temporalcharacteristics in addition to different start and stop times may beused such that some projected features 904 are illuminatedsimultaneously (with differing temporal characteristics). For example,the discussed temporal characteristics of multiple on and off events ofdiffering numbers, duration, intensity, frequency, etc. may be usedsimultaneously (in time) at various projected features 904. In additionor in the alternative, the same temporal characteristics at the sametime may be used at projected features 904 that are aligned verticallyalong the y-axis (e.g., orthogonal to an epipolar axis) as discussedherein.

FIG. 11 illustrates an example illumination scheme 1100 for use infeature matching, arranged in accordance with at least someimplementations of the present disclosure. As will be appreciated, anyillumination scheme discussed herein is implemented by pixels of dynamicprojector 105. Such illumination schemes are presented with respect toscene 900 for the sake of clarity of presentation; however suchillumination schemes are implemented by pixels of dynamic projector 105at scale. For example, each projected feature provided by anillumination scheme is generated using a pixel of dynamic projector 105.Furthermore, in the following, scene 900 is shown with elements thereofremoved and projected features are shown as black dots for the sake ofclarity of presentation. As shown in FIG. 11, illumination scheme 1100includes providing, for a row 1101 of projected features 1104, the sameillumination intensity signal simultaneously. As discussed, a first axis(e.g., horizontal, x-axis) may be aligned with the feature matchingwhile a second axis (e.g., vertical, y-axis) is orthogonal thereto. Suchan orientation may be exploited by providing the same illuminationintensity signal (having particular temporal characteristics)simultaneously such that row 1101 of projected features 1104 (and alsopixels of dynamic projector 105) is orthogonal to an epipolar axis oftriangulation.

As shown, in an embodiment, the illumination intensity signal (havingparticular temporal characteristics) is scanned to a next row (orcolumn) of projected features 1104 (and also pixels of dynamic projector105) in a scan direction 1105 over time. For example, projected features1104 of row 1101 are illuminated in a first time frame using theparticular temporal characteristics. Then, in a subsequent time frame,another row of projected features 1104 are illuminated using theparticular temporal characteristics, and so on across scene 900.

FIG. 12 illustrates another example illumination scheme 1200 for use infeature matching, arranged in accordance with at least someimplementations of the present disclosure. As shown in FIG. 12,illumination scheme 1200 includes providing, for a first row 1201 ofprojected features 1204 and a second row 1202 of projected features1205, different illumination intensity signals 1211, 1212 such thatillumination intensity signals 1211, 1212 overlap at least partially.For example, illumination intensity signal 1211 may be implemented byeach projected feature 1204 of first row 1201 and illumination intensitysignal 1212 may be implemented by each projected feature 1205 of firstrow 1202. As shown with respect to timing diagram 1210, illuminationintensity signal 1211 has temporal characteristics including a firston/off illumination and a second on/off illumination while illuminationintensity signal 1212 has temporal characteristics including a singleon/off illumination that corresponds temporally to the first on/offillumination of illumination intensity signal 1211. As discussed withrespect to FIG. 12, rows 1201, 1202 may have the same illuminationintensity signals 1211, 1212 within the rows since feature matching isnot performed along the y-axis. Furthermore, the differing temporalcharacteristics of illumination intensity signals 1211, 1212 allow forfeature matching along the x-axis. For example, a pixel of a dynamicvision sensor (when matching sensor to projector) or pixels of multipledynamic vision sensors (when matching between sensors) are matched basedon a response to illumination intensity signal 1211 or a response toillumination intensity signal 1212 such that the feature matching isdifferentiated based on the different differing temporal characteristicsof illumination intensity signals 1211, 1212.

For example, a pixel of a dynamic vision sensor responsive toillumination intensity signal 1211 would have an ON event at time to, anOFF event at time t1, an ON event at time t2, and an OFF event at timet3. In contrast, a pixel of a dynamic vision sensor responsive toillumination intensity signal 1212 would have an ON event at time t0 andan OFF event at time t1 without any events at times t2, t3. Suchillumination intensity signals 1211, 1212 may be varied using anysuitable technique or techniques such as differing illumination starttimes, differing illumination durations, differing illumination endtimes, different numbers of illumination events, different illuminationfrequencies, etc. As used herein, the term illumination start timeindicates a start of a projected feature being turned on or ramped up toan on state, an illumination duration indicates an projected featurebeing held in an on state (whether at the same or varying intensity), anillumination end time indicates a projected feature being turned off orramped to an off state, an illumination event indicates a projectedfeature being turned on and off or otherwise cycled between a high andlow illumination intensity, and an illumination frequency indicates afrequency between start times, end times, or the like.

As discussed, differing temporal characteristics such as those discussedwith respect to illumination intensity signals 1211, 1212 may be thesame along a first axis (e.g., e.g., vertically along the y-axis) anddifferent along a second axis (e.g., horizontally along the x-axis). Inother embodiments, differing temporal characteristics are provided atother locations across or around a scene (as provided by pixels of adynamic projector). Furthermore, in some embodiments, the same temporalcharacteristics may be provided along an epipolar axis so long as theyare separated by a sufficient distance to allow feature matching.

FIG. 13 illustrates yet another example illumination scheme 1300 for usein feature matching, arranged in accordance with at least someimplementations of the present disclosure. As shown in FIG. 13,illumination scheme 1300 includes providing, for a first row 1301 ofprojected features 1304 and a second row 1302 of projected features1305, the same illumination intensity signals 1311, 1312. Also as shown,projected features 1304 and projected features 1305 are separated by adistance d. For example, matching illumination intensity signals 1311,1312 may be implemented by each projected feature 1304 of first row 1301and each projected feature 1305 of first row 1302 so long as projectedfeatures 1304 and projected features 1305 are separated by at least adistance d (and pixels of the dynamic projector are separated by adisparity, disp) such that matching illumination intensity signals 1311,1312 do not hinder feature matching as discussed herein. The distance,d, and/or dynamic projector pixel disparity, disp, may be any suitablevalues. For example, the pixel disparity may be one-fourth, or one-halfof a total pixel disparity of a maximum width (in pixels) of the dynamicprojector.

As shown with respect to timing diagram 1310, illumination intensitysignals 1311, 1312 match such that both have temporal characteristicsincluding a first on/off illumination and a second on/off illumination.As discussed, the lateral distance between projected features 1304, 1305implementing illumination intensity signals 1311, 1312 may provide forfeature matching as discussed herein. It is noted that such repeating ofillumination intensity signals 1311, 1312 may have an associated cost ofmistakes in pixel matches with a corresponding benefit of increasedspeed.

FIG. 14 illustrates an example device 1400 for determiningcorrespondence between a dynamic vision sensor and/or between multipledynamic vision sensors, arranged in accordance with at least someimplementations of the present disclosure. As shown in FIG. 14, device1400 may include dynamic vision camera 104, dynamic vision camera 201,dynamic projector 105, and a motherboard 1401 to implement, within ahousing 1402 of device 1400, memory 107, ISP 101, and computer visionmodule 106. Also as shown, device 1400 may include a display port 1403to transmit image data for presentment to a user via a display (notshown), which may be implemented as an integrated component of device1400 or separately from device 1400.

With reference to FIGS. 1 and 2, in some embodiments, feature matchingmodule 102 may be implemented via hardware (e.g., a graphics processoror ISP 101) to generate depth map 113 or a depth image based on featurematching between dynamic vision cameras 104, 201 and/or feature matchingbetween dynamic projector 105 and one or both of dynamic vision cameras104, 201 using techniques discussed herein. As shown, dynamic visioncameras 104, 201 and, optionally, dynamic projector 105 may behorizontally aligned or substantially horizontally aligned with respectto scene 121 to perform feature matching along a horizontal axis orplane.

FIG. 15 is a flow diagram illustrating an example process 1500 forcorrelating a feature detected at a dynamic vision sensor pixel to afeature of a projected light pattern, arranged in accordance with atleast some implementations of the present disclosure. Process 1500 mayinclude one or more operations 1501-1503 as illustrated in FIG. 15.Process 1500 may form at least part of a dynamic vision sensor pixel toprojected feature matching process. By way of non-limiting example,process 1500 may be performed by any device, system, or combinationthereof as discussed herein. Furthermore, process 1500 will be describedherein with reference to system 1600 of FIG. 16, which may perform oneor more operations of process 1500.

FIG. 16 is an illustrative diagram of an example system 1600 forcorrelating a feature detected at a dynamic vision sensor pixel to afeature of a projected light pattern, arranged in accordance with atleast some implementations of the present disclosure. As shown in FIG.16, system 1600 may include a central processor 1601, a graphicsprocessor 1602, a memory 1603, dynamic vision cameras 104, 201, dynamicprojector 105, driver 108, and ISP 101. Also as shown, central processor1601 may include or implement computer vision module 106 and ISP 101 mayinclude or implement feature matching module 102 and light patternmodule 103. In the example of system 1600, memory 1603 may store pixelsignal data, control signal data, depth map data, pixel correspondencedata, and/or any other data as discussed herein.

As shown, in some embodiments, computer vision module 106 is implementedby central processor 1601 and feature matching module 102 and lightpattern module 103 are implemented by ISP 101. In some embodiments,computer vision module 106 is implemented by ISP 101 or graphicsprocessor 1602. In some embodiments, one or both of feature matchingmodule 102 and light pattern module 103 are implemented by centralprocessor 1601 or graphics processor 1602.

Graphics processor 1602 may include any number and type of graphicsprocessing units that may provide or implement operations discussedherein. For example, graphics processor 1602 may include circuitrydedicated to manipulate pixel signal data or the like obtained frommemory 1603. ISP 101 may include any number and type of image signal orimage processing units that may provide the discussed feature matching,depth map generation, and computer vision operations and/or otheroperations as discussed herein. For example, ISP 101 may includecircuitry dedicated to pixel signal data and/or light pattern data suchas an ASIC or the like. Central processor 1601 may include any numberand type of processing units or modules that may provide control andother high level functions for system 1600 and/or provide the discussedfeature matching, depth map generation, and computer vision operationsand/or other operations as discussed herein. Memory 1603 may be any typeof memory such as volatile memory (e.g., Static Random Access Memory(SRAM), Dynamic Random Access Memory (DRAM), etc.) or non-volatilememory (e.g., flash memory, etc.), and so forth. In a non-limitingexample, memory 1603 may be implemented by cache memory.

In an embodiment, one or more or portions of feature matching module102, light pattern module 103, and computer vision module 106 may beimplemented via an execution unit (EU) of ISP 101 or graphics processor1602. The EU may include, for example, programmable logic or circuitrysuch as a logic core or cores that may provide a wide array ofprogrammable logic functions. In an embodiment, one or more or portionsof feature matching module 102, light pattern module 103, and computervision module 106 may be implemented via dedicated hardware such asfixed function circuitry or the like of ISP 101 or graphics processor1602. Fixed function circuitry may include dedicated logic or circuitryand may provide a set of fixed function entry points that may map to thededicated logic for a fixed purpose or function.

Returning to discussion of FIG. 15, process 1500 begins at operation1501, where a light pattern is cast onto a scene such that the lightpattern includes multiple projected features having differing temporalcharacteristics. In an embodiment, dynamic projector 105 casts, onto thescene, the light pattern including multiple projected features such thatthe projected features have differing temporal characteristics. Thediffering temporal characteristics may include any temporalcharacteristics discussed herein such as illumination start time, endtime, or duration, a number and/or frequency of illumination events,etc. Furthermore the difference may be any suitable difference such asstart time difference, end time difference, duration difference, anumber of illumination events difference, frequency difference, etc. Thediffering temporal characteristics include any number of differingtemporal characteristics and/or differing temporal characteristicssignatures (e.g., each temporal characteristics signature includingmultiple temporal characteristics) that differ between any number ofprojected features. As discussed herein, such projected featurescorrespond to pixel locations of dynamic projector 105 such that pixelsprovide the discussed differing temporal characteristics between pixelsthereof. For example, dynamic projector 105 may be considered a reversedcamera that provides projected features from pixels thereof (incomparison to a camera that receives projected features at pixelsthereof).

In an embodiment, the light pattern includes a first projected featureor component from a first pixel of dynamic projector 105 and a secondprojected feature or component from a second pixel of dynamic projector105 such that the first and second projected feature or component haveshared temporal characteristics and the first pixel and the second pixelare orthogonal to an epipolar axis between dynamic projector 105 anddynamic vision sensor 104. In an embodiment, the light pattern includesa first projected feature or component projected at a first angle fromdynamic projector 105 and a second projected feature or componentprojected at a second angle from dynamic projector 105 such that thefirst and second projected feature or component have shared temporalcharacteristics and the first and the second angles are orthogonal tothe epipolar axis between dynamic projector 105 and dynamic visionsensor 104. For example, the first and second projected feature orcomponent may have the same temporal characteristics (e.g.,substantially identical temporal characteristics with differences onlybetween the tolerance capabilities of dynamic projector 105). Asdiscussed, providing shared temporal characteristics orthogonal to theepipolar axis does not hinder pixel matching along the epipolar axis andmay instead increase performance as multiple pixel matches are performedsimultaneously.

In an embodiment, the light pattern includes a first projected featureor component from a first pixel of dynamic projector 105 and a secondprojected feature or component from a second pixel of dynamic projector105 such that the first and second projected features or components havefirst and second temporal characteristic, respectively, that overlaptemporally but differ in at least one of an emission duration, anemission count, or an emission frequency. Due to the difference intemporal characteristics (despite the overlap), the first pixel and thesecond pixel may have any orientation therebetween. For example, thefirst pixel and the second pixel may be aligned with an epipolar axisbetween dynamic projector 105 and dynamic vision sensor 104.

As discussed, dynamic projector 105 may be any suitable dynamicprojector having any suitable components such that dynamic projector 105may cast a light pattern having multiple projected features withdiffering temporal characteristics. In an embodiment, dynamic projector105 includes multiple light emitters (e.g., VCSELs) each correspondingto a component of the projected features. In an embodiment, the dynamicprojector comprises a plurality of light emitters and one or moreoptical elements over the light emitters, the one or more opticalelements to split light emitted from each of the light emitters intocorresponding patterns of projected features. In an embodiment, dynamicprojector 105 includes an edge laser and a scanning mirror to cast thelight pattern onto the scene. For example, the edge laser and a scanningmirror may work in concert to generate each component of the projectedfeatures. Furthermore, dynamic projector 105 may be driven by anysuitable circuitry. In an embodiment, projector driver 108 is coupled todynamic projector 105 and one or more of ISP 101, central processor1601, and graphics processor 1602 and the projector driver is to providea signal to dynamic projector 105 and one or more of ISP 101, centralprocessor 1601, and graphics processor 1602 such that the signal isindicative of the light pattern. For example, dynamic projector 105 mayuse the signal to generate the light pattern and one or more of ISP 101,central processor 1601, and graphics processor 1602 may use the signalfor matching purposes.

Processing continues at operation 1502, where a signal may be generatedthat indicates changes in detected luminance for pixels of dynamicvision sensor 104. In an embodiment, dynamic vision sensor 104 generatesa pixel signal indicating changes in detected luminance for pixels ofdynamic vision sensor 104. For example, the pixel signal may includeidentifiers of ON or OFF events, time stamps for such events, acorresponding pixel for such events, and other information as needed.For example, for an individual pixel of dynamic vision sensor 104, thepixel signal may provide an indication only when there is a change inillumination intensity at the individual pixel. In this sense dynamicvision sensor 104 acts as a passive device that detects changes withoutproactively generating an image frame or the like (as in the case of atraditional sensor). Dynamic vision sensor 104 (and/or dynamic visionsensor 201) may include any suitable components that provide forpixel-by-pixel detection of illumination intensity changes. In anembodiment, a pixel of dynamic vision sensor 104 and/or dynamic visionsensor 201 includes a photoreceptor, a differencing circuit, and acomparator, the differencing circuit and comparator to detect anincrease or decrease in photocurrent of the photoreceptor.

Processing continues at operation 1503, where, based on the signalgenerated at operation 1502, a feature detected by the dynamic visionsensor is determined that corresponds to an individual projected featureof the multiple projected features based on the signal indicating achange in detected luminance corresponding to an individual temporalcharacteristic of the individual projected feature. In an embodiment, anindividual pixel of the dynamic vision sensor that corresponds to anindividual projected feature of the light pattern is determined based onthe individual pixel having a change in detected luminance correspondingto an individual temporal characteristic of the individual projectedfeature. For example, a pixel or pixels of dynamic projector 105 maygenerate a corresponding projected feature having particular orindividual temporal characteristics that are unique to thepixel/projected feature. A pixel of dynamic vision sensor 104 senses theprojected feature due to being spatially aligned therewith and generatesa signal corresponding to the individual temporal characteristics. Forexample, the pixel signal may include ON and OFF events as discussedherein. The pixel is then correlated to the projected feature based onthe match of the temporal characteristics of the pixel signal and thetemporal characteristics of the projected feature as discussed herein.

Such a correlation may be used in any suitable application or imagingcontext. In an embodiment, a depth value for a depth map of the scene isdetermined based on a location of the individual pixel of dynamic visionsensor 104 and a projected angle of that projected the individualprojected feature or component of the light pattern. In an embodiment, adepth value for a depth map of the scene is determined based on theindividual pixel of dynamic vision sensor 104 and the dynamic projectorpixel of dynamic projector 105 that projected the individual componentof the light pattern. Such a depth value may be generated using anysuitable technique or techniques such as triangulation techniques. Inaddition or in the alternative, a second dynamic vision sensor 201 maygenerate a second signal indicating second changes in detected luminancefor second pixels of the second dynamic vision sensor and a secondindividual pixel of the second dynamic vision sensor 201 correspondingto the individual projected feature of the light pattern may bedetermined based on the second individual pixel having a second changein detected luminance corresponding to the individual temporalcharacteristic of the individual projected feature in analogy withoperations 1502, 1503 as discussed. A depth value may then be determinedbased on the location of the individual pixel in dynamic vision sensor104 and the location of second individual pixel in second dynamic visionsensor 201 using any suitable technique or techniques such astriangulation techniques.

Process 1500 may be repeated any number of times either in series or inparallel for any number pixels or the like. For example, process 1500may provide for a dynamic vision sensor pixel to projected featurematching process for a plurality of projected features of a lightpattern cast onto a scene. In an embodiment, process 1500 furtherincludes generating a depth map or a disparity map based on such pixelmatching.

Various components of the systems described herein may be implemented insoftware, firmware, and/or hardware and/or any combination thereof. Forexample, various components of the systems discussed herein may beprovided, at least in part, by hardware of a computing System-on-a-Chip(SoC) such as may be found in a computing system such as, for example, asmartphone. Those skilled in the art may recognize that systemsdescribed herein may include additional components that have not beendepicted in the corresponding figures. For example, the systemsdiscussed herein may include additional components such ascommunications modules and the like that have not been depicted in theinterest of clarity.

While implementation of the example processes discussed herein mayinclude the undertaking of all operations shown in the orderillustrated, the present disclosure is not limited in this regard and,in various examples, implementation of the example processes herein mayinclude only a subset of the operations shown, operations performed in adifferent order than illustrated, or additional operations.

In addition, any one or more of the operations discussed herein may beundertaken in response to instructions provided by one or more computerprogram products. Such program products may include signal bearing mediaproviding instructions that, when executed by, for example, a processor,may provide the functionality described herein. The computer programproducts may be provided in any form of one or more machine-readablemedia. Thus, for example, a processor including one or more graphicsprocessing unit(s) or processor core(s) may undertake one or more of theblocks of the example processes herein in response to program codeand/or instructions or instruction sets conveyed to the processor by oneor more machine-readable media. In general, a machine-readable mediummay convey software in the form of program code and/or instructions orinstruction sets that may cause any of the devices and/or systemsdescribed herein to implement at least portions of the systems discussedherein or any other module or component as discussed herein.

As used in any implementation described herein, the term “module” or“component” refers to any combination of software logic, firmware logic,hardware logic, and/or circuitry configured to provide the functionalitydescribed herein. The software may be embodied as a software package,code and/or instruction set or instructions, and “hardware”, as used inany implementation described herein, may include, for example, singly orin any combination, hardwired circuitry, programmable circuitry, statemachine circuitry, fixed function circuitry, execution unit circuitry,and/or firmware that stores instructions executed by programmablecircuitry. The modules may, collectively or individually, be embodied ascircuitry that forms part of a larger system, for example, an integratedcircuit (IC), system on-chip (SoC), and so forth.

FIG. 17 is an illustrative diagram of an example system 1700, arrangedin accordance with at least some implementations of the presentdisclosure. In various implementations, system 1700 may be a mobilesystem although system 1700 is not limited to this context. System 1700may implement and/or perform any modules or techniques discussed herein.For example, system 1700 may be incorporated into a personal computer(PC), sever, laptop computer, ultra-laptop computer, tablet, touch pad,portable computer, handheld computer, palmtop computer, personal digitalassistant (PDA), cellular telephone, combination cellular telephone/PDA,television, smart device (e.g., smartphone, smart tablet or smarttelevision), mobile internet device (MID), messaging device, datacommunication device, cameras (e.g. point-and-shoot cameras, super-zoomcameras, digital single-lens reflex (DSLR) cameras), and so forth. Insome examples, system 1700 may be implemented via a cloud computingenvironment.

In various implementations, system 1700 includes a platform 1702 coupledto a display 1720. Platform 1702 may receive content from a contentdevice such as content services device(s) 1730 or content deliverydevice(s) 1740 or other similar content sources. A navigation controller1750 including one or more navigation features may be used to interactwith, for example, platform 1702 and/or display 1720. Each of thesecomponents is described in greater detail below.

In various implementations, platform 1702 may include any combination ofa chipset 1705, processor 1710, memory 1712, antenna 1713, storage 1714,graphics subsystem 1715, applications 1716 and/or radio 1718. Chipset1705 may provide intercommunication among processor 1710, memory 1712,storage 1714, graphics subsystem 1715, applications 1716 and/or radio1718. For example, chipset 1705 may include a storage adapter (notdepicted) capable of providing intercommunication with storage 1714.

Processor 1710 may be implemented as a Complex Instruction Set Computer(CISC) or Reduced Instruction Set Computer (RISC) processors, x86instruction set compatible processors, multi-core, or any othermicroprocessor or central processing unit (CPU). In variousimplementations, processor 1710 may be dual-core processor(s), dual-coremobile processor(s), and so forth.

Memory 1712 may be implemented as a volatile memory device such as, butnot limited to, a Random Access Memory (RAM), Dynamic Random AccessMemory (DRAM), or Static RAM (SRAM).

Storage 1714 may be implemented as a non-volatile storage device suchas, but not limited to, a magnetic disk drive, optical disk drive, tapedrive, an internal storage device, an attached storage device, flashmemory, battery backed-up SDRAM (synchronous DRAM), and/or a networkaccessible storage device. In various implementations, storage 1714 mayinclude technology to increase the storage performance enhancedprotection for valuable digital media when multiple hard drives areincluded, for example.

Graphics subsystem 1715 may perform processing of images such as stillor video for display. Graphics subsystem 1715 may be a graphicsprocessing unit (GPU) or a visual processing unit (VPU), for example. Ananalog or digital interface may be used to communicatively couplegraphics subsystem 1715 and display 1720. For example, the interface maybe any of a High-Definition Multimedia Interface, DisplayPort, wirelessHDMI, and/or wireless HD compliant techniques. Graphics subsystem 1715may be integrated into processor 1710 or chipset 1705. In someimplementations, graphics subsystem 1715 may be a stand-alone devicecommunicatively coupled to chipset 1705.

The graphics and/or video processing techniques described herein may beimplemented in various hardware architectures. For example, graphicsand/or video functionality may be integrated within a chipset.Alternatively, a discrete graphics and/or video processor may be used.As still another implementation, the graphics and/or video functions maybe provided by a general purpose processor, including a multi-coreprocessor. In further embodiments, the functions may be implemented in aconsumer electronics device.

Radio 1718 may include one or more radios capable of transmitting andreceiving signals using various suitable wireless communicationstechniques. Such techniques may involve communications across one ormore wireless networks. Example wireless networks include (but are notlimited to) wireless local area networks (WLANs), wireless personal areanetworks (WPANs), wireless metropolitan area network (WMANs), cellularnetworks, and satellite networks. In communicating across such networks,radio 1718 may operate in accordance with one or more applicablestandards in any version.

In various implementations, display 1720 may include any television typemonitor or display. Display 1720 may include, for example, a computerdisplay screen, touch screen display, video monitor, television-likedevice, and/or a television. Display 1720 may be digital and/or analog.In various implementations, display 1720 may be a holographic display.Also, display 1720 may be a transparent surface that may receive avisual projection. Such projections may convey various forms ofinformation, images, and/or objects. For example, such projections maybe a visual overlay for a mobile augmented reality (MAR) application.Under the control of one or more software applications 1716, platform1702 may display user interface 1722 on display 1720.

In various implementations, content services device(s) 1730 may behosted by any national, international and/or independent service andthus accessible to platform 1702 via the Internet, for example. Contentservices device(s) 1730 may be coupled to platform 1702 and/or todisplay 1720. Platform 1702 and/or content services device(s) 1730 maybe coupled to a network 1760 to communicate (e.g., send and/or receive)media information to and from network 1760. Content delivery device(s)1740 also may be coupled to platform 1702 and/or to display 1720.

In various implementations, content services device(s) 1730 may includea cable television box, personal computer, network, telephone, Internetenabled devices or appliance capable of delivering digital informationand/or content, and any other similar device capable ofuni-directionally or bi-directionally communicating content betweencontent providers and platform 1702 and/display 1720, via network 1760or directly. It will be appreciated that the content may be communicateduni-directionally and/or bi-directionally to and from any one of thecomponents in system 1700 and a content provider via network 1760.Examples of content may include any media information including, forexample, video, music, medical and gaming information, and so forth.

Content services device(s) 1730 may receive content such as cabletelevision programming including media information, digital information,and/or other content. Examples of content providers may include anycable or satellite television or radio or Internet content providers.The provided examples are not meant to limit implementations inaccordance with the present disclosure in any way.

In various implementations, platform 1702 may receive control signalsfrom navigation controller 1750 having one or more navigation features.The navigation features of navigation controller 1750 may be used tointeract with user interface 1722, for example. In various embodiments,navigation controller 1750 may be a pointing device that may be acomputer hardware component (specifically, a human interface device)that allows a user to input spatial (e.g., continuous andmulti-dimensional) data into a computer. Many systems such as graphicaluser interfaces (GUI), and televisions and monitors allow the user tocontrol and provide data to the computer or television using physicalgestures.

Movements of the navigation features of navigation controller 1750 maybe replicated on a display (e.g., display 1720) by movements of apointer, cursor, focus ring, or other visual indicators displayed on thedisplay. For example, under the control of software applications 1716,the navigation features located on navigation controller 1750 may bemapped to virtual navigation features displayed on user interface 1722,for example. In various embodiments, navigation controller 1750 may notbe a separate component but may be integrated into platform 1702 and/ordisplay 1720. The present disclosure, however, is not limited to theelements or in the context shown or described herein.

In various implementations, drivers (not shown) may include technologyto enable users to instantly turn on and off platform 1702 like atelevision with the touch of a button after initial boot-up, whenenabled, for example. Program logic may allow platform 1702 to streamcontent to media adaptors or other content services device(s) 1730 orcontent delivery device(s) 1740 even when the platform is turned “off”In addition, chipset 1705 may include hardware and/or software supportfor 5.1 surround sound audio and/or high definition 7.1 surround soundaudio, for example. Drivers may include a graphics driver for integratedgraphics platforms. In various embodiments, the graphics driver mayinclude a peripheral component interconnect (PCI) Express graphics card.

In various implementations, any one or more of the components shown insystem 1700 may be integrated. For example, platform 1702 and contentservices device(s) 1730 may be integrated, or platform 1702 and contentdelivery device(s) 1740 may be integrated, or platform 1702, contentservices device(s) 1730, and content delivery device(s) 1740 may beintegrated, for example. In various embodiments, platform 1702 anddisplay 1720 may be an integrated unit. Display 1720 and content servicedevice(s) 1730 may be integrated, or display 1720 and content deliverydevice(s) 1740 may be integrated, for example. These examples are notmeant to limit the present disclosure.

In various embodiments, system 1700 may be implemented as a wirelesssystem, a wired system, or a combination of both. When implemented as awireless system, system 1700 may include components and interfacessuitable for communicating over a wireless shared media, such as one ormore antennas, transmitters, receivers, transceivers, amplifiers,filters, control logic, and so forth. An example of wireless sharedmedia may include portions of a wireless spectrum, such as the RFspectrum and so forth. When implemented as a wired system, system 1700may include components and interfaces suitable for communicating overwired communications media, such as input/output (I/O) adapters,physical connectors to connect the I/O adapter with a correspondingwired communications medium, a network interface card (NIC), disccontroller, video controller, audio controller, and the like. Examplesof wired communications media may include a wire, cable, metal leads,printed circuit board (PCB), backplane, switch fabric, semiconductormaterial, twisted-pair wire, co-axial cable, fiber optics, and so forth.

Platform 1702 may establish one or more logical or physical channels tocommunicate information. The information may include media informationand control information. Media information may refer to any datarepresenting content meant for a user. Examples of content may include,for example, data from a voice conversation, videoconference, streamingvideo, electronic mail (“email”) message, voice mail message,alphanumeric symbols, graphics, image, video, text and so forth. Datafrom a voice conversation may be, for example, speech information,silence periods, background noise, comfort noise, tones and so forth.Control information may refer to any data representing commands,instructions or control words meant for an automated system. Forexample, control information may be used to route media informationthrough a system, or instruct a node to process the media information ina predetermined manner. The embodiments, however, are not limited to theelements or in the context shown or described in FIG. 17.

As described above, system 1700 may be embodied in varying physicalstyles or form factors. FIG. 18 illustrates an example small form factordevice 1800, arranged in accordance with at least some implementationsof the present disclosure. In some examples, system 1700 may beimplemented via device 1800. In other examples, other systems discussedherein or portions thereof may be implemented via device 1800. Invarious embodiments, for example, device 1800 may be implemented as amobile computing device a having wireless capabilities. A mobilecomputing device may refer to any device having a processing system anda mobile power source or supply, such as one or more batteries, forexample.

Examples of a mobile computing device may include a personal computer(PC), laptop computer, ultra-laptop computer, tablet, touch pad,portable computer, handheld computer, palmtop computer, personal digitalassistant (PDA), cellular telephone, combination cellular telephone/PDA,smart device (e.g., smartphone, smart tablet or smart mobiletelevision), mobile internet device (MID), messaging device, datacommunication device, cameras (e.g. point-and-shoot cameras, super-zoomcameras, digital single-lens reflex (DSLR) cameras), and so forth.

Examples of a mobile computing device also may include computers thatare arranged to be worn by a person, such as a wrist computers, fingercomputers, ring computers, eyeglass computers, belt-clip computers,arm-band computers, shoe computers, clothing computers, and otherwearable computers. In various embodiments, for example, a mobilecomputing device may be implemented as a smartphone capable of executingcomputer applications, as well as voice communications and/or datacommunications. Although some embodiments may be described with a mobilecomputing device implemented as a smartphone by way of example, it maybe appreciated that other embodiments may be implemented using otherwireless mobile computing devices as well. The embodiments are notlimited in this context.

As shown in FIG. 18, device 1800 may include a housing with a front 1801and a back 1802. Device 1800 includes a display 1804, an input/output(I/O) device 1806, color camera 104, color camera 105, infraredtransmitter 204, and an integrated antenna 1808. Device 1800 also mayinclude navigation features 1812. I/O device 1806 may include anysuitable I/O device for entering information into a mobile computingdevice. Examples for I/O device 1806 may include an alphanumerickeyboard, a numeric keypad, a touch pad, input keys, buttons, switches,microphones, speakers, voice recognition device and software, and soforth. Information also may be entered into device 1800 by way ofmicrophone (not shown), or may be digitized by a voice recognitiondevice. As shown, device 1800 may include color cameras 104, 105 and aflash 1810 integrated into back 1802 (or elsewhere) of device 1800. Inother examples, color cameras 104, 105 and flash 1810 may be integratedinto front 1801 of device 1800 or both front and back sets of camerasmay be provided. Color cameras 104, 105 and a flash 1810 may becomponents of a camera module to originate color image data with IRtexture correction that may be processed into an image or streamingvideo that is output to display 1804 and/or communicated remotely fromdevice 1800 via antenna 1808 for example.

Various embodiments may be implemented using hardware elements, softwareelements, or a combination of both. Examples of hardware elements mayinclude processors, microprocessors, circuits, circuit elements (e.g.,transistors, resistors, capacitors, inductors, and so forth), integratedcircuits, application specific integrated circuits (ASIC), programmablelogic devices (PLD), digital signal processors (DSP), field programmablegate array (FPGA), logic gates, registers, semiconductor device, chips,microchips, chip sets, and so forth. Examples of software may includesoftware components, programs, applications, computer programs,application programs, system programs, machine programs, operatingsystem software, middleware, firmware, software modules, routines,subroutines, functions, methods, procedures, software interfaces,application program interfaces (API), instruction sets, computing code,computer code, code segments, computer code segments, words, values,symbols, or any combination thereof. Determining whether an embodimentis implemented using hardware elements and/or software elements may varyin accordance with any number of factors, such as desired computationalrate, power levels, heat tolerances, processing cycle budget, input datarates, output data rates, memory resources, data bus speeds and otherdesign or performance constraints.

One or more aspects of at least one embodiment may be implemented byrepresentative instructions stored on a machine-readable medium whichrepresents various logic within the processor, which when read by amachine causes the machine to fabricate logic to perform the techniquesdescribed herein. Such representations, known as IP cores may be storedon a tangible, machine readable medium and supplied to various customersor manufacturing facilities to load into the fabrication machines thatactually make the logic or processor.

While certain features set forth herein have been described withreference to various implementations, this description is not intendedto be construed in a limiting sense. Hence, various modifications of theimplementations described herein, as well as other implementations,which are apparent to persons skilled in the art to which the presentdisclosure pertains are deemed to lie within the spirit and scope of thepresent disclosure.

In one or more first embodiments, a dynamic projector to cast, onto ascene, a light pattern comprising multiple projected features, theprojected features comprising differing temporal characteristics, adynamic vision sensor to generate a signal indicating changes indetected luminance for pixels of the dynamic vision sensor, and aprocessor coupled to the dynamic vision sensor, the processor to receivethe signal and to determine a feature detected by the dynamic visionsensor corresponds to an individual projected feature of the multipleprojected features based on the signal indicating a change in detectedluminance corresponding to an individual temporal characteristic of theindividual projected feature.

In one or more second embodiments, further to the first embodiments, theprocessor is further to identify at least one pixel of the dynamicvision sensor corresponding to the feature detected by the dynamicvision sensor.

In one or more third embodiments, further to any of the first or secondembodiments, the processor is further to determine a depth value for adepth map of the scene based on a location of the at least one pixel inthe dynamic vision sensor and a projected angle corresponding to theindividual projected feature of the light pattern.

In one or more fourth embodiments, further to any of the first throughthird embodiments, the device further comprises a second dynamic visionsensor to generate a second signal indicating second changes in detectedluminance for second pixels of the second dynamic vision sensor, whereinthe processor is further to determine a second feature detected by thesecond dynamic vision sensor corresponds to the individual projectedfeature based on the second signal indicating a second change indetected luminance corresponding to the individual temporalcharacteristic of the individual projected feature.

In one or more fifth embodiments, further to any of the first throughfourth embodiments, the processor is further to determine a depth valuefor a depth map of the scene based on a location of a pixel of thedynamic vision sensor corresponding to the feature detected by thedynamic vision sensor and a location of a pixel of the second dynamicvision sensor corresponding to the second feature detected by the seconddynamic vision sensor.

In one or more sixth embodiments, further to any of the first throughfifth embodiments, the dynamic projector, the dynamic vision sensor, andthe second dynamic vision sensor are configured in a triangular shapesuch that the dynamic projector is off-axis and orthogonal to an axisbetween the dynamic vision sensor and the second dynamic vision sensor.

In one or more seventh embodiments, further to any of the first throughsixth embodiments, the light pattern comprises a first projected featurefrom a first pixel of the dynamic projector and a second projectedfeature from a second pixel of the dynamic projector, the first andsecond projected features having shared temporal characteristics,wherein the first pixel and the second pixel are orthogonal to anepipolar axis between the dynamic projector and the dynamic visionsensor.

In one or more eighth embodiments, further to any of the first throughseventh embodiments, the light pattern comprises a first projectedfeature from a first pixel of the dynamic projector and a secondprojected feature from a second pixel of the dynamic projector, thefirst and second projected features having first and second temporalcharacteristics, respectively, wherein the first and second temporalcharacteristic overlap temporally but differ in at least one of anemission duration, an emission count, or an emission frequency.

In one or more ninth embodiments, further to any of the first througheighth embodiments, the dynamic projector comprises a plurality of lightemitters each corresponding to a projected feature of the projectedfeatures.

In one or more tenth embodiments, further to any of the first throughninth embodiments, the dynamic projector comprises a plurality of lightemitters and one or more optical elements over the light emitters, theone or more optical elements to split light emitted from each of thelight emitters into corresponding patterns of projected features.

In one or more eleventh embodiments, further to any of the first throughtenth embodiments, the dynamic projector comprises an edge laser and ascanning mirror to cast the light pattern onto the scene.

In one or more twelfth embodiments, further to any of the first througheleventh embodiments, a pixel of the dynamic vision sensor comprises aphotoreceptor, a differencing circuit, and a comparator, thedifferencing circuit and comparator to detect an increase or decrease inphotocurrent of the photoreceptor.

In one or more thirteenth embodiments, further to any of the firstthrough twelfth embodiments, the device further comprises a projectordriver coupled to the dynamic projector and the processor, the projectordriver to provide a signal to the dynamic projector and the processor,the signal indicative of the light pattern.

In one or more fourteenth embodiments, a method comprises casting, ontoa scene, a light pattern comprising multiple projected features, theprojected features comprising differing temporal characteristics,generating a signal indicating changes in detected luminance for pixelsof a dynamic vision sensor, and determining, based on the signal, a setof pixels of the dynamic vision sensor corresponding to an individualprojected feature of the multiple projected features based on the set ofpixels having a change in detected luminance corresponding to anindividual temporal characteristic of the individual projected feature.

In one or more fifteenth embodiments, further to the fourteenthembodiments, the method further comprises determining a depth value fora depth map of the scene based on pixel locations of the set of pixelscorresponding to the individual projected feature and a projected anglecorresponding to the individual projected feature of the light pattern.

In one or more sixteenth embodiments, further to any of the fourteenthor fifteenth embodiments, the method further comprises generating asecond signal indicating second changes in detected luminance for secondpixels of the second dynamic vision sensor and determining a second setof pixels of the second dynamic vision sensor corresponding to theindividual projected feature based on the second set of pixels having asecond change in detected luminance corresponding to the individualtemporal characteristic of the individual projected feature.

In one or more seventeenth embodiments, further to any of the fourteenththrough sixteenth embodiments, the method further comprises determininga depth value for a depth map of the scene based on pixel locations ofthe set of pixels in the dynamic vision sensor and second pixellocations of the second set of pixels in the second dynamic visionsensor.

In one or more eighteenth embodiments, further to any of the fourteenththrough sixteenth embodiments, the light pattern comprises a firstprojected feature from a first angle of the dynamic projector and asecond projected feature from a second angle of the dynamic projector,the first and second projected features having shared temporalcharacteristics, wherein the first angle and the second angle areorthogonal to an epipolar axis between the dynamic projector and thedynamic vision sensor.

In one or more nineteenth embodiments, further to any of the fourteenththrough eighteenth embodiments, the light pattern comprises a firstprojected feature from a first angle of the dynamic projector and asecond projected feature from a second angle of the dynamic projector,the first and second projected features having first and second temporalcharacteristics, respectively, wherein the first and second temporalcharacteristics overlap temporally but differ in at least one of anemission duration, an emission count, or an emission frequency.

In one or more twentieth embodiments, at least one machine readablemedium may include a plurality of instructions that in response to beingexecuted on a computing device, causes the computing device to perform amethod according to any one of the above embodiments.

In one or more twenty-first embodiments, an apparatus may include meansfor performing a method according to any one of the above embodiments.

It will be recognized that the embodiments are not limited to theembodiments so described, but can be practiced with modification andalteration without departing from the scope of the appended claims. Forexample, the above embodiments may include specific combination offeatures. However, the above embodiments are not limited in this regardand, in various implementations, the above embodiments may include theundertaking only a subset of such features, undertaking a differentorder of such features, undertaking a different combination of suchfeatures, and/or undertaking additional features than those featuresexplicitly listed. The scope of the embodiments should, therefore, bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

What is claimed is:
 1. A device comprising: a dynamic projector to cast,onto a scene, a light pattern comprising multiple projected features,the projected features comprising differing temporal characteristics,the multiple projected features comprising a first projected featurehaving a first turn on time, a subsequent first turn off time, asubsequent second turn on time, and a subsequent second turn off time,and a second projected feature having a third turn on time and asubsequent third turn off time, wherein the third turn on time or thethird turn off time is temporally between the first turn on time and thesecond turn off time; a dynamic vision sensor to generate a plurality ofsignals, for each of a plurality of pixels of the dynamic vision sensor,indicating temporal changes in detected luminance temporally inclusiveof the differing temporal characteristics of the multiple projectedfeatures; and a processor coupled to the dynamic vision sensor, theprocessor to receive the signals and to determine a first pixel of thedynamic vision sensor corresponds to the first projected feature basedon a first signal of the plurality of signals indicating changes indetected luminance corresponding to the first and second turn on timesand the first and second turn off times and to determine a second pixelof the dynamic vision sensor corresponds to the second projected featurebased on a second signal of the plurality of signals indicating changesin detected luminance corresponding to the third turn on time and thethird turn off time.
 2. The device of claim 1, wherein the processor isfurther to determine a depth value for a depth map of the scene based ona location of the first pixel of the dynamic vision sensor and aprojected angle corresponding to the first projected feature.
 3. Thedevice of claim 1, further comprising: a second dynamic vision sensor togenerate a second plurality of signals indicating second temporalchanges in detected luminance for second pixels of the second dynamicvision sensor, wherein the processor is further to: determine a thirdpixel of the second dynamic vision sensor corresponds to the firstprojected feature based on a third signal of the second plurality ofsignals indicating a change in detected luminance corresponding to thefirst projected feature.
 4. The device of claim 3, wherein the processoris further to determine a depth value for a depth map of the scene basedon a location of the first pixel of the dynamic vision sensor and alocation of the third pixel of the second dynamic vision sensor.
 5. Thedevice of claim 3, wherein the dynamic projector, the dynamic visionsensor, and the second dynamic vision sensor are configured in atriangular shape such that the dynamic projector is off-axis andorthogonal to an axis between the dynamic vision sensor and the seconddynamic vision sensor.
 6. The device of claim 1, wherein the firstprojected feature is from a third pixel of the dynamic projector and athird projected feature is from a fourth pixel of the dynamic projector,the third projected feature having the first turn on time, the firstturn off time, the second turn on time, and the second turn off time,wherein the third pixel and the fourth pixel are orthogonal to anepipolar axis between the dynamic projector and the dynamic visionsensor.
 7. The device of claim 1, wherein the first turn on time and thethird turn on time match and the first turn off time and the third turnoff time match.
 8. The device of claim 1, wherein the dynamic projectorcomprises a plurality of light emitters each corresponding to aprojected feature of the projected features.
 9. The device of claim 1,wherein the dynamic projector comprises a plurality of light emittersand one or more optical elements over the light emitters, the one ormore optical elements to split light emitted from each of the lightemitters into corresponding patterns of projected features.
 10. Thedevice of claim 1, wherein the dynamic projector comprises an edge laserand a scanning mirror to cast the light pattern onto the scene.
 11. Thedevice of claim 1, wherein a pixel of the dynamic vision sensorcomprises a photoreceptor, a differencing circuit, and a comparator, thedifferencing circuit and comparator to detect an increase or decrease inphotocurrent of the photoreceptor.
 12. The device of claim 1, furthercomprising: a projector driver coupled to the dynamic projector and theprocessor, the projector driver to provide a control signal to thedynamic projector and the processor, the control signal indicative ofthe light pattern.
 13. A method comprising: casting, onto a scene, alight pattern comprising multiple projected features, the projectedfeatures comprising differing temporal characteristics, the multipleprojected features comprising a first projected feature having a firstturn on time, a subsequent first turn off time, a subsequent second turnon time, and a subsequent second turn off time, and a second projectedfeature having a third turn on time and a subsequent third turn offtime, wherein the third turn on time or the third turn off time istemporally between the first turn on time and the second turn off time;generating a plurality of signals, for each of a plurality of pixels ofa dynamic vision sensor, indicating temporal changes in detectedluminance temporally inclusive of the differing temporal characteristicsof the multiple projected features; and determining, based on thesignals, a first pixel of the dynamic vision sensor corresponding to thefirst projected feature based on a first signal of the plurality ofsignals indicating changes in detected luminance corresponding to thefirst and second turn on times and the first and second turn off timesand a second pixel of the dynamic vision sensor corresponding to thesecond projected feature based on a second signal of the plurality ofsignals indicating changes in detected luminance corresponding to thethird turn on time and the third turn off time.
 14. The method of claim13, further comprising: determining a depth value for a depth map of thescene based on a location of the first pixel of the dynamic visionsensor and a projected angle corresponding to the first projectedfeature.
 15. The method of claim 13, further comprising: generating asecond plurality of signals indicating second temporal changes indetected luminance for second pixels of the second dynamic visionsensor; and determining a third pixel of the second dynamic visionsensor corresponding to the first projected feature based on a thirdsignal of the second plurality of signals indicating a change indetected luminance corresponding to the first projected feature.
 16. Themethod of claim 15, further comprising: determining a depth value for adepth map of the scene based on a location of the first pixel of thedynamic vision sensor and a location of the third pixel of the seconddynamic vision sensor.
 17. The method of claim 13, wherein the firstprojected feature is from a first angle of the dynamic projector and athird projected feature is from a second angle of the dynamic projector,the third projected feature having the first turn on time, the firstturn off time, the second turn on time, and the second turn off time,wherein the first angle and the second angle are orthogonal to anepipolar axis between the dynamic projector and the dynamic visionsensor.
 18. The method of claim 13, wherein the first turn on time andthe third turn on time match and the first turn off time and the thirdturn off time match.
 19. At least one non-transitory machine readablemedium comprising a plurality of instructions that, in response to beingexecuted on a device, cause the device to: cast, onto a scene, a lightpattern comprising multiple projected features, the projected featurescomprising differing temporal characteristics, the multiple projectedfeatures comprising a first projected feature having a first turn ontime, a subsequent first turn off time, a subsequent second turn ontime, and a subsequent second turn off time, and a second projectedfeature having a third turn on time and a subsequent third turn offtime, wherein the third turn on time or the third turn off time istemporally between the first turn on time and the second turn off time;generate a plurality of signals, for each of a plurality of pixels of adynamic vision sensor, indicating temporal changes in detected luminancetemporally inclusive of the differing temporal characteristics of themultiple projected features; and determine, based on the signals, afirst pixel of the dynamic vision sensor corresponding to the firstprojected feature based on a first signal of the plurality of signalsindicating changes in detected luminance corresponding to the first andsecond turn on times and the first and second turn off times and asecond pixel of the dynamic vision sensor corresponding to the secondprojected feature based on a second signal of the plurality of signalsindicating changes in detected luminance corresponding to the third turnon time and the third turn off time.
 20. The non-transitory machinereadable medium of claim 19, the machine readable medium comprisingfurther instructions that, in response to being executed on the device,cause the device to: determine a depth value for a depth map of thescene based on a location of the first pixel of the dynamic visionsensor and a projected angle corresponding to the first projectedfeature.
 21. The non-transitory machine readable medium of claim 19, themachine readable medium comprising further instructions that, inresponse to being executed on the device, cause the device to: generatea second plurality of signals indicating second temporal changes indetected luminance for second pixels of the second dynamic visionsensor; and determine a third pixel of the second dynamic vision sensorcorresponding to the first projected feature based on a third signal ofthe second plurality of signals indicating a change in detectedluminance corresponding to the first projected feature.
 22. Thenon-transitory machine readable medium of claim 21, the machine readablemedium comprising further instructions that, in response to beingexecuted on the device, cause the device to: determine a depth value fora depth map of the scene based on a location of the first pixel of thedynamic vision sensor and a location of the third pixel of the seconddynamic vision sensor.
 23. The non-transitory machine readable medium ofclaim 19, wherein the first projected feature is from a first angle ofthe dynamic projector and a third projected feature is from a secondangle of the dynamic projector, the third projected feature having thefirst turn on time, the first turn off time, the second turn on time,and the second turn off time, wherein the first angle and the secondangle are orthogonal to an epipolar axis between the dynamic projectorand the dynamic vision sensor.
 24. The non-transitory machine readablemedium of claim 19, wherein the first turn on time and the third turn ontime match and the first turn off time and the third turn off timematch.